Methods and systems for assigning pixels distance-cost values using a flood fill technique

ABSTRACT

Disclosed herein are methods and systems for assigning pixels distance-cost values using a flood fill technique. One embodiment takes the form of a process that includes obtaining video data depicting a head of a user, obtaining depth data associated with the video data, and selecting seed pixels for a flood fill at least in part by using the depth information. The process also includes performing the flood fill from the selected seed pixels. The flood fill assigns respective distance-cost values to pixels of the video data based on position-space cost values and color-space cost values. In some embodiments, the process also includes classifying pixels of the video data as foreground based at least in part on the assigned distance-cost values. In some other embodiments, the process also includes assigning pixels of the video data foreground-likelihood values based at least in part on the assigned distance-cost values.

BACKGROUND

Online data communications are quite prevalent and pervasive in modernsociety, and are becoming more so all the time. Moreover, developmentsin software, communication protocols, and peripheral devices (e.g.,video cameras, three-dimension video cameras, and the like), along withdevelopments in other computing disciplines, have collectively enabledand facilitated the inclusion of multimedia experiences as part of suchcommunications. Indeed, the multimedia nature and aspects of a givencommunication session are often the focus and even essence of suchcommunications. These multimedia experiences take forms such as audiochats, video chats (that are usually also audio chats), online meetings(e.g., web meetings), and of course many other examples could be listedas well.

Using the context of online meetings as an illustrative example, it isoften the case that one of the participants in the video conference callis a designated presenter, and often this user opts to embed a digitalrepresentation of themselves (i.e., a persona) as part of the offeredpresentation. By way of example, the user may choose to have a videofeed embedded into a power point presentation. In a simple scenario, thevideo feed may include a depiction of the user as well as backgroundinformation. The background information may include a view of the wallbehind the user as seen from the point of view of the video camera. Ifthe user is outside, the background information may include buildingsand trees. In more advanced versions of this video conferencingparadigm, the persona is isolated from the background information foundin video feed. This allows viewers to experience a more naturalsensation as the embedded persona they see within the presentation isnot cluttered and surrounded by distracting and undesired backgroundinformation.

Overview

Improvements over the above-described developments have recently beenrealized by technology that, among other capabilities and features,extracts what is known as a “persona” of a user from a video feed from avideo camera that is capturing video of the user. The extracted persona,which in some examples appears as a depiction of part of the user (i.e.,upper torso, shoulders, arms, hands, neck, and head) and in otherexamples appears as a depiction of the entire user. This technology isdescribed in the following patent documents, each of which isincorporated in its respective entirety into this disclosure: (i) U.S.patent application Ser. No. 13/083,470, entitled “Systems and Methodsfor Accurate User Foreground Video Extraction,” filed Apr. 8, 2011 andpublished Oct. 13, 2011 as U.S. Patent Application Pub. No.US2011/0249190, (ii) U.S. patent application Ser. No. 13/076,264,entitled “Systems and Methods for Embedding a Foreground Video into aBackground Feed based on a Control Input,” filed Mar. 30, 2011 andpublished Oct. 6, 2011 as U.S. Patent Application Pub. No.US2011/0242277, and (iii) unpublished U.S. patent application Ser. No.14/145,874, entitled “System and Methods for Persona IdentificationUsing Combined Probability Maps,” filed Dec. 31, 2013.

Facilitating accurate and precise extraction of the persona, especiallythe hair of the persona, from a video feed is not a trivial matter. Atleast one aspect of some user extraction processes includes classifyingpixels as foreground of the video data. In some processes classificationpixels as background is performed. As mentioned, persona extraction iscarried out with respect to video data that is received from a camerathat is capturing video of a scene in which the user is positioned. Thepersona-extraction technology substantially continuously (e.g., withrespect to each frame) identifies which pixels represent the user (i.e.,the foreground) and which pixels do not (i.e., the background), andaccordingly generates “alpha masks” (e.g., generates an alpha mask foreach frame), where a given alpha mask may take the form of or at leastinclude an array with a respective stored data element corresponding toeach pixel in the corresponding frame, where such stored data elementsare individually and respectively set equal to 1 (one) for each userpixel (a.k.a. for each foreground pixel) and to 0 (zero) for every otherpixel (i.e., for each non-user (a.k.a. background) pixel).

The described alpha masks correspond in name with the definition of the“A” in the “RGBA” pixel-data format known to those of skill in the art,where “R” is a red-color value, “G” is a green-color value, “B” is ablue-color value, and “A” is an alpha value ranging from 0 (completetransparency) to 1 (complete opacity). In a typical implementation, the“0” in the previous sentence may take the form of a hexadecimal numbersuch as 0x00 (equal to a decimal value of 0 (zero)), while the “1” maytake the form of a hexadecimal number such as 0xFF (equal to a decimalvalue of 255); that is, a given alpha value may be expressed as an 8-bitnumber that can be set equal to any integer that is (i) greater than orequal to zero and (ii) less than or equal to 255. Moreover, a typicalRGBA implementation provides for such an 8-bit alpha number for each ofwhat are known as the red channel, the green channel, and the bluechannel; as such, each pixel has (i) a red (“R”) color value whosecorresponding transparency value can be set to any integer value between0x00 and 0xFF, (ii) a green (“G”) color value whose correspondingtransparency value can be set to any integer value between 0x00 and0xFF, and (iii) a blue (“B”) color value whose correspondingtransparency value can be set to any integer value between 0x00 and0xFF. And certainly other pixel-data formats could be used, as deemedsuitable by those having skill in the relevant art for a givenimplementation.

When merging an extracted persona with content, the above-referencedpersona-based technology creates the above-mentioned merged display in amanner consistent with these conventions; in particular, on apixel-by-pixel (i.e., pixel-wise) basis, the merging is carried outusing pixels from the captured video frame for which the correspondingalpha-mask values equal 1, and otherwise using pixels from the content.Moreover, it is noted that pixel data structures typically also includeor are otherwise associated with one or more other values correspondingrespectively to one or more other properties of the pixel, wherebrightness is an example of one such property. In some embodiments, thebrightness value is the luma component of the image or video frame. Inother embodiments, the brightness value is the pixel values of one of anR, G, or B color channel, or other similar color space (e.g., gammacompressed RGB, or R′G′B′, or YUV, or YCbCr, as examples). In otherembodiments, the brightness value may be a weighted average of pixelvalues from one or more color channels. And other approaches exist aswell.

This disclosure describes systems and methods for assigning pixelsdistance-cost values using a flood fill technique. Such systems andmethods are useful for, among other things, scenarios in which a user'spersona is to be extracted from a video feed, for example, in an online“panel discussion” or more generally an online meeting or other onlinecommunication session. The present systems and methods facilitatenatural interaction by providing the accurate and precise identificationof the user's hair, a particularly troublesome aspect of a comprehensiveuser extraction process. The present systems and methods thereforeprovide an advanced approach for assigning pixels distance-cost valuesusing a flood fill technique, which may in turn be used to classifypixels as foreground or background in the context of a personaextraction process. Such a classification may take the form of a hard(e.g., binary) classification or a soft (e.g., probabilistic)classification.

One embodiment of the systems and methods disclosed herein takes theform of a process. The process includes obtaining video data depicting ahead of a user. The process also includes obtaining depth dataassociated with the video data. The process also includes selecting seedpixels for a flood fill at least in part by using the depth information.The process also includes performing the flood fill from the selectedseed pixels. The flood fill assigns respective distance-cost values topixels of the video data based on position-space cost values andcolor-space cost values.

Another embodiment takes the form of a system that includes acommunication interface, a processor, and data storage containinginstructions executable by the processor for causing the system to carryout at least the functions described in the preceding paragraph.

Moreover, any of the variations and permutations described in theensuing paragraphs and anywhere else in this disclosure can beimplemented with respect to any embodiments, including with respect toany method embodiments and with respect to any system embodiments.Furthermore, this flexibility and cross-applicability of embodiments ispresent in spite of the use of slightly different language (e.g.,process, method, steps, functions, set of functions, and the like) todescribe and or characterize such embodiments.

In at least one embodiment, obtaining the video data includes obtainingthe video data using a video camera. The video camera may be athree-dimension (3-D) video camera that captures the video data as wellas the depth data associated with the video data. In such an embodiment,obtaining depth data associated with the video data includes obtainingthe depth data via the 3-D video camera. In another embodiment,obtaining the video data includes obtaining the video data via a datastore.

In at least one embodiment, obtaining depth data associated with thevideo data includes obtaining depth data associated with the video datausing one or more of a depth sensor, a depth camera, a 3-D video camera,and a light field camera. In embodiments wherein a light field camera isemployed, both the video data and the depth data are obtained via thelight field camera. In another embodiment, obtaining depth data includesobtaining depth data via a data store.

Obtaining the video data and obtaining the depth data may includeobtaining the video data from a first camera and obtaining the depthdata from a second camera. Obtaining the video data and obtaining thedepth data may include obtaining the video data from a first camera andobtaining the depth data from the first camera as well. Obtaining thevideo data and obtaining the depth data may include obtaining the videodata from a first set of cameras and obtaining the depth data from asecond set of cameras. Obtaining the video data and obtaining the depthdata may include obtaining the video data from a first set of camerasand obtaining the depth data from the first set of cameras. Obtainingthe video data and obtaining the depth data may include obtaining thevideo data from a first set of cameras and obtaining the depth data froma subset of the first set of cameras. Obtaining the video data andobtaining the depth data may include obtaining the depth data from afirst set of cameras and obtaining the video data from a subset of thefirst set of cameras.

In at least one embodiment, selecting seed pixels for the flood fillfurther includes selecting seed pixels for the flood fill at least inpart by using the video data.

In at least one embodiment, selecting seed pixels for the flood fillincludes (i) obtaining an upper contour of a head of the user, and (ii)selecting pixels on the obtained upper contour as seed pixels for theflood fill. Such an embodiment is referred to in the balance of thisdisclosure as an upper contour embodiment.

In at least one upper contour embodiment, obtaining the upper contour ofthe head of the user includes generating the upper contour of the headof the user based at least in part on the depth data associated with thevideo data. In at least one such embodiment, generating the uppercontour of the head of the user based at least in part on a thresholddepth value.

In at least one upper contour embodiment, obtaining the upper contour ofthe head of the user comprises (i) obtaining a head contour thatestimates an outline of the head of the user, and (ii) identifying anupper portion of the obtained head contour as being the upper contour ofthe head of the user. The head contour may be obtained at least in partusing depth data associated with the video data. As another example, thehead contour may be obtained via facial recognition techniques which arewell known by those with skill in the relevant art.

In some upper contour embodiments, the selected seed pixels are equallydistributed along the upper contour. In other upper contour embodiments,the selected seed pixels are not equally distributed along the uppercontour. In at least one upper contour embodiment, the selected seedpixels include every pixel on the upper contour. In at least one uppercontour embodiment, the selected seed pixels do not include every pixelon the upper contour.

In at least one upper contour embodiment, the selected seed pixels areof colors that are found in a user-hair-color model. In at least onefurther upper contour embodiment, the selected seed pixels are of colorsthat are found in the user-hair color model at least a threshold numberof times. In some embodiments, the selected seed pixels are of colorsthat are above a threshold level of certainty of being a user-hair coloraccording to a user-hair-color model.

In at least one upper contour embodiment, the selected seed pixels areof colors that are not found in a background-color model. In at leastone further upper contour embodiment, the selected seed pixels are ofcolors that are found in the background-color model less than athreshold number of times. In some embodiment, the selected seed pixelsare of colors that are below a threshold level of certainty of being abackground color according to a background-color model.

In at least one embodiment, selecting seed pixels for the flood fillincludes (i) identifying noisy depth-pixels within the obtained depthdata, and (ii) selecting the identified noisy depth-pixels as seedpixels for the flood fill. The noisy depth-pixels have intermittentdepth values. Such an embodiment is referred to in the balance of thisdisclosure as a noisy depth-pixel embodiment.

In at least one noisy depth-pixel embodiment, the selected seed pixelsare located within an extended head box.

In at least one noisy depth-pixel embodiment, the selected seed pixelshave intermittent depth values similar to a depth value corresponding tothe head of the user (i.e., the selected seed pixels have intermittentdepth values that are within a threshold tolerance of a depth valuecorresponding to the head of the user).

In at least one noisy depth-pixel embodiment, the selected seed pixelsare of colors that are found in a user-hair-color model. In at least onenoisy depth-pixel embodiment, the selected seed pixels are not of colorsthat are found in a background-color model.

In at least one noisy depth-pixel embodiment, the selected seed pixelsare of colors that are found a user-hair color model at least athreshold number of times. In at least one noisy depth-pixel embodiment,the selected seed pixels are of colors that are not found abackground-hair color model more than a threshold number of times.

In some noisy depth-pixel embodiments, the selected seed pixels are ofcolors that are above a threshold level of certainty of being auser-hair color according to a user-hair-color model. In some noisydepth-pixel embodiment, the selected seed pixels are of colors that arebelow a threshold level of certainty of being a background coloraccording to a background-color model.

In at least one embodiment, the process further includes initializingthe distance-cost values of the seed pixels to be zero.

In at least one embodiment, a first set of the selected seed pixels areon an upper contour and a second set of the selected seed pixels haveintermittent depth values. In at least one such embodiment, the processfurther includes initializing the distance-cost values of the seedpixels in the first set to be zero and initializing the distance-costvalues of the seed pixels in the second set to be non-zero.

In at least one embodiment, the distance-cost value of a given pixelincludes (i) a position-space cost value from a seed pixel to the givenpixel and (ii) a color-space cost value from the seed pixel to the givenpixel.

In at least one embodiment, the distance-cost value of a given pixel isa geodesic-distance-cost value from a seed pixel to the given pixel. Thegeodesic-distance-cost value is a combination of a position-space costvalue from the seed pixel to the given pixel and (ii) a color-space costvalue from the seed pixel to the given pixel.

In at least one embodiment, the distance-cost value of a given pixelincludes (i) a position-space cost value from a seed pixel to the givenpixel and (ii) a summation of color-space step-cost values along a floodfill path from the seed pixel to the given pixel.

In at least one embodiment, the distance-cost value of a given pixelincludes (i) a position-space cost value from a seed pixel to the givenpixel and (ii) a color-space cost value that is based at least in parton a user-hair color model and a color of the given pixel.

In at least one embodiment, performing the flood fill includes (i)identifying a plurality of neighbor pixels of a current pixel, (ii)determining respective step-cost values from the current pixel to eachpixel in the plurality of neighbor pixels, and (iii) assigning eachpixel in the plurality of neighbor pixels a respective distance-costvalue based on a distance-cost value of the current pixel and therespective step-cost values.

In at least one embodiment, performing the flood fill includes (i)determining a minimum distance-cost value from at least one of theselected seed pixels to a current pixel, and (ii) assigning the currentpixel a distance-cost value that is the determined minimum distance-costvalue.

In at least one embodiment, determining a minimum distance-cost valueincludes comparing a current distance-cost value corresponding with acurrent flood fill path to a prior distance-cost value correspondingwith a prior flood fill path.

In at least one such embodiment, the current flood fill path and theprior flood fill path originate from a common seed pixel. In at leastone other such embodiment, the current flood fill path and the priorflood fill path originate from different seed pixels.

In at least one embodiment, performing the flood fill comprisesperforming the flood fill along a plurality of flood-fill paths. In atleast one such embodiment the process further includes terminating theflood fill along a current flood-fill path in response to at least onetermination criteria. The termination criteria includes a current pixelnot being a user-hair color according to a user-hair-color model, thecurrent pixel being a background color according to a background-colormodel, a distance-cost value of the current pixel being greater than adistance-cost threshold, and a step-cost value of the current pixelbeing greater than a step-cost threshold. Of course many othertermination criteria could be employed as well such as aposition-space-cost value of the current pixel being greater than aposition-space-cost threshold and a color-space-cost value of thecurrent pixel being greater than a color-space-cost threshold.

In at least one embodiment, the process further includes classifyingpixels of the video data as foreground based at least in part on theassigned distance-cost values. In at least one embodiment, the processfurther includes classifying pixels of the video data as backgroundbased at least in part on the assigned distance-cost values. In at leastone embodiment, the process further includes assigning pixels of thevideo data foreground-likelihood values based at least in part on theassigned distance-cost values. In at least one embodiment, the processfurther includes assigning pixels of the video databackground-likelihood values based at least in part on the assigneddistance-cost values.

At a high level, the systems and processes described herein use videodata and novel processing techniques to assign distance-cost values topixels of the video data. The assigned distance-cost values may be usedto classify pixels of the video data as foreground. The video datadepicts a head of a user. The user may or may not have hair on the topof their head. The systems and processes described herein may be used toidentify the hair of the user and in turn classify the hair of the useras foreground. The identified hair may in turn be used as part of acomprehensive user extraction (foreground identification) process. Partof identifying the hair of the user involves performing a flood fill toassign distance-cost values to pixels of the video data.

Depending on the nature of the obtained video data and the obtaineddepth data, pixels may take on a plurality of forms.

In scenarios wherein a single frame of information includes both videodata and depth data, pixels in such a frame include both colorinformation and depth information. In such a scenario the term depthpixel references the depth information of a pixel and the terms pixel ofvideo data, color pixel, and the like reference the color information ofthe pixel. In such a scenario the term pixel may be used to referenceeither or both the color information and the depth information. Ofcourse, any pixel has an associated location and even when notexplicated stated this would be well known by those with skill in theart.

In scenarios wherein there are separate frames of video data and depthdata there is a correspondence between the frames of video data and theframes of depth data. Therefore, if a depth pixel is identified within aframe of depth data it is evident that a corresponding pixel of videodata may be included within that identification and vice versa.

Seed pixels, along with video data, are inputs to a flood fill process.The flood fill process assigns pixels of video data distance-costvalues. A distance-cost value is a value that grows larger as thelikelihood that a pixel is part of the user's hair decreases. Thedistance-cost value of a given pixel is based on a position-space costvalue associated with a seed pixel and the given pixel and a color-spacecost value associated with either the seed pixel, previous-step pixel,or a user-hair-color model and the given pixel. A hair-likelihood (orforeground-likelihood) may be based on an assigned distance-cost valueof the given pixel.

A user-hair-color model and a background-color model may each take on aplurality of forms. In general each model is used to indicate whichcolors are representative of a user-hair color and a background of thevideo data respectively. The models may take on the form of a histogram,a Gaussian mixture, an array of color values and respective colorcounts, and the like.

The flood fill process may be a recursive process. In one embodiment, ateach pixel, the flood fill will identify a set of nearby pixels andassign to each of them respective distance-cost values. Then this samemethod will be done with respect to each of those nearby pixels. Thiswill happen many times, until assigned distance-cost values (or certainparts of assigned distance-cost values) reach a threshold or until otherflood fill termination criteria are met. Of course, as this may beimplemented as a massively parallel process, pixels will be visited morethan once as a result of the use of various different flood fill paths(i.e., series of steps to different nearby pixels).

In general, any indication, classification, assignment, and the like ofpixels, regions, portions, and the like of the video data is relevantwithin the scope of the systems and processes described herein. As thisdisclosure describes systems and processes that may be used as part of acomprehensive user-extraction process, it is explicitly noted that it isnot required that any classification of pixels as foreground orbackground be definitive with respect to the entire user-extractionprocess.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The accompanying figures, where like reference numerals refer toidentical or functionally similar elements throughout the separateviews, together with the detailed description below, are incorporated inand form part of the specification, and serve to further illustrateembodiments of concepts that include the claimed invention, and explainvarious principles and advantages of those embodiments.

FIG. 1 depicts an example process, in accordance with an embodiment.

FIG. 2 depicts seed pixels on an upper contour, in accordance with anembodiment.

FIG. 3 depicts a generation of an upper contour using depth data, inaccordance with an embodiment.

FIG. 4 depicts a first example generation of an upper contour using ahead contour, in accordance with an embodiment.

FIG. 5 depicts a second example generation of an upper contour using ahead contour, in accordance with an embodiment.

FIG. 6 depicts noisy depth-pixels, in accordance with an embodiment.

FIG. 7 depicts a first set of seed pixels selected from the noisydepth-pixels of FIG. 6, in accordance with an embodiment.

FIG. 8 depicts a second set of seed pixels selected from the noisydepth-pixels of FIG. 6, in accordance with an embodiment.

FIG. 9 depicts a third set of seed pixels selected from the noisydepth-pixels of FIG. 6, in accordance with an embodiment.

FIG. 10 depicts an array of pixels including a current pixel and a setof neighbor pixels in a first flood fill step, in accordance with anembodiment.

FIG. 11 depicts an array of pixels including a current pixel and a setof neighbor pixels in a second flood fill step, in accordance with anembodiment.

FIG. 12 depicts an array of pixels including a current pixel and a setof neighbor pixels in a third flood fill step, in accordance with anembodiment.

FIG. 13 depicts an array of pixels including two seed pixels and a givenpixel, in accordance with an embodiment.

FIG. 14 depicts an example computing and communication device (CCD), inaccordance with an embodiment.

Skilled artisans will appreciate that elements in the figures areillustrated for simplicity and clarity and have not necessarily beendrawn to scale. For example, the dimensions of some of the elements inthe figures may be exaggerated relative to other elements to help toimprove understanding of embodiments of the present invention.

The apparatus and method components have been represented whereappropriate by conventional symbols in the drawings, showing only thosespecific details that are pertinent to understanding the embodiments ofthe present invention so as not to obscure the disclosure with detailsthat will be readily apparent to those of ordinary skill in the arthaving the benefit of the description herein.

DETAILED DESCRIPTION OF THE INVENTION

Before proceeding with this detailed description, it is noted that theentities, connections, arrangements, and the like that are depictedin—and described in connection with—the various figures are presented byway of example and not by way of limitation. As such, any and allstatements or other indications as to what a particular figure“depicts,” what a particular element or entity in a particular figure“is” or “has,” and any and all similar statements—that may in isolationand out of context be read as absolute and therefore limiting—can onlyproperly be read as being constructively preceded by a clause such as“In at least one embodiment, . . . ” And it is for reasons akin tobrevity and clarity of presentation that this implied leading clause isnot repeated ad nauseum in this detailed description.

FIG. 1 depicts an example process, in accordance with at least oneembodiment. In particular, FIG. 1 depicts an example process 100 thatincludes elements 102-108. Although primarily depicted and described asbeing performed serially, at least a portion of the elements (steps) ofthe process 100 may be performed contemporaneously, or in a differentorder than is depicted in and described in connection with FIG. 1.Furthermore, in at least one embodiment, the process 100 is repeated, atsome point in time, after any of the elements 102-108 are completed.Additionally, repetition of the process 100 may or may not includeperformance of each element in the process 100, and may commence at anyof the elements 102-108. The process 100 is further described below.

One embodiment takes the form of the process 100. The process 100includes obtaining video data depicting a head of a user. The process100 also includes obtaining depth data associated with the video data.The process 100 also includes selecting seed pixels for a flood fill atleast in part by using the depth data. The process 100 also includesperforming the flood fill from the selected seed pixels. The flood fillassigns respective distance-cost values to pixels of the video databased on position-space cost values and color-space cost values.

At element 102 the process 100 includes obtaining video data depicting ahead of a user.

At element 104 the process 100 includes obtaining depth data associatedwith the obtained video data.

At element 106 the process 100 includes using at least the obtaineddepth data to select seed points for a flood fill.

At element 108 the process 100 includes assigning pixels of the obtainedvideo data respective distance-cost values through performance of theflood fill that takes as input the selected seed points. The respectivedistance-cost values are based at least in part on respectiveposition-space cost values and respective color-space cost values.

In at least one embodiment, the process 100 further includes classifyingpixels of the video data as foreground based at least in part on theassigned distance-cost values. In at least one embodiment, the process100 further includes classifying pixels of the video data as backgroundbased at least in part on the assigned distance-cost values. In at leastone embodiment, the process 100 further includes assigning pixels of thevideo data foreground-likelihood values based at least in part on theassigned distance-cost values. In at least one embodiment, the process100 further includes assigning pixels of the video databackground-likelihood values based at least in part on the assigneddistance-cost values.

FIG. 2 depicts seed pixels on an upper contour, in accordance with anembodiment. In particular, FIG. 2 depicts a graphical overview 200 thatincludes depth data 202, a head contour 204, an upper contour 206, andseed pixels 208. The depth data 202 may represent a first frame of depthinformation. The depth data 202 may be generated via a depth camera, 3-Dcamera, or the like and may be obtained via a communication interface.The depth data 202 may alternatively be obtained via a data store. Thedepth data 202 may correspond with one or more frames of video data. Amapping of depth-pixels to color-pixels within the frames may beincluded. Alternatively, each pixel in a frame of obtained video datamay include color and depth information inherently as describedpreviously in the Overview. It follows that a frame of 3-D data mayeither correspond with a single array of pixels, wherein each pixelincludes both depth and color information or two arrays of pixels,wherein each pixel in the first array includes depth information andeach pixel in the second array includes color information.

Described generally, in at least one embodiment, selecting the seedpixels 208 for the flood fill includes (i) obtaining the upper contour206 of the head of the user, and (ii) selecting pixels on the obtainedupper contour 206 as the seed pixels 208 for the flood fill. In at leastone embodiment, the process further includes initializing thedistance-cost values of the seed pixels to be zero. In at least oneembodiment, obtaining the upper contour 206 of the head of the userincludes generating the upper contour 206 of the head of the user basedat least in part on the depth data 202 associated with the video data.In at least one such embodiment, generating the upper contour 206 of thehead of the user is based at least in part on a threshold depth value. Afurther description of one example of this embodiment is discussed inrelation with FIG. 3.

In at least one embodiment, obtaining the upper contour of the head ofthe user comprises (i) obtaining the head contour 204 that estimates anoutline of the head of the user, and (ii) identifying an upper portionof the obtained head contour as being the upper contour 206 of the headof the user. The head contour 204 may be obtained at least in part usingdepth data 202 associated with the video data. As another example, thehead contour 204 may be obtained via facial recognition techniques whichare well known by those with skill in the relevant art. A furtherdescription of one example of this step is discussed in relation withFIGS. 4&5.

In some embodiments, the selected seed pixels 208 are equallydistributed along the upper contour 206. In other embodiments, theselected seed pixels 208 are not equally distributed along the uppercontour 206. In at least one embodiment, the selected seed pixels 208include every pixel on the upper contour 206 and in at least one otherembodiment, the selected seed pixels 208 do not include every pixel onthe upper contour 206.

Additionally, in some embodiments, selecting the seed pixels 208 for theflood fill further includes selecting the seed pixels 208 for the floodfill at least in part by using the obtained video data. In someembodiments, selecting the seed pixels 208 for the flood fill furtherincludes selecting the seed pixels 208 for the flood fill at least inpart by using color information. The color information may come in theform of a user-hair-color model. The user-hair-color model is aconstruct that estimates which color values (e.g., RGB color values) arepart of the user's hair and how often each color value is depicted inthe video data. When selecting the seed pixels 208 it is desirable thateach seed pixel is a color that is found in the user's hair. In at leastone embodiment, the selected seed pixels 208 are of colors that arefound in a user-hair-color model. In at least one further embodiment,the selected seed pixels 208 are of colors are found the user-hair colormodel at least a threshold number of times. This further embodimentallows for statistically reliable results. It may be the case that a setof potential seed pixels is selected prior to selecting the seed pixels208 and only those pixels in the set of potential seed pixels that havedesirable color values (e.g., colors found in the user-hair-color model)are selected as the seed pixels 208.

Additionally, in some embodiments, the color information may come in theform of a background-color model. The background-color model is aconstruct that estimates which color values (e.g., RGB color values) arepart of the background (normally the part of the background borderingthe user's hair) and how often each color value is depicted in the videodata. When selecting the seed pixels 208 it is sometimes desirable thateach seed pixel is not a color that is found in the background. In atleast one embodiment, the selected seed pixels 208 are of colors thatare found in a background-color model. In at least one furtherembodiment, the selected seed pixels 208 are of colors are found thebackground-color model at least a threshold number of times. Thisfurther embodiment allows for statistically reliable results. It may bethe case that a set of potential seed pixels is selected prior toselecting the seed pixels 208 and only those pixels in the set ofpotential seed pixels that have desirable color values (e.g., colors notfound in the background-color model) are selected as the seed pixels208.

FIG. 3 depicts a generation of an upper contour using depth data, inaccordance with an embodiment. In particular, FIG. 3 depicts aconceptual overview 300 that includes depth data 302 a, depth data 302 bwith an identified region of threshold depth 304, and depth data 302 cwith an upper contour 306. The depth data 302 a-c may represent a firstframe of depth information. The depth data 302 a-c may be generated viaa depth camera, 3-D camera, or the like and obtained via a communicationinterface. The depth data 302 a-c may alternatively be obtained via adata store. The depth data 302 b and 302 c depict the same depthinformation as the depth data 302 a, but additionally show the thresholddepth 304 and the upper contour 306 respectively. FIG. 3 may be used asan example of generating the upper contour 206 of FIG. 2 based at leastin part on a threshold depth value. In such an example, the depth data302 a-c is equivalent to the depth data 202 of FIG. 2 and the uppercontour 306 is equivalent to the upper contour 206 of FIG. 2.

In at least one embodiment, wherein generating the upper contour 306 isbased at least in part on depth data (i.e., the depth data 302 a orequivalently the depth data 302 b and 302 c) associated with the videodata, generating the upper contour 306 is based at least in part on thethreshold depth 304. The threshold depth 304 is employed to helpidentify a region of the depth data 302 a-c (and therefore, acorresponding region in the associated video data) wherein thedepth-pixels within the identified region have respective depth valuesthat fall within the threshold depth 304.

As depicted in the conceptual overview 300, a region is identified,wherein the region corresponds with depth values that fall within thethreshold depth 304. The threshold depth 304 may be a single value or arange of values. The threshold depth 304 may indicate a region of depthinformation that is greater than a particular depth value, less than aparticular depth value, included within a range of depth values, orexcluded from a range of depth values. For example, the threshold depth304 could correspond to a region with depth values greater than 20 cmfrom a depth camera but less than 35 cm from the depth camera.

In some embodiments, the threshold depth 304 is a set value, or range ofvalues, that is obtained via a data store. It is immutable and ishard-coded into the systems and processes described herein. In someother embodiments, the threshold depth 304 is generated through ananalysis of the depth data 302 a-c. For example, a sub-process canidentify for a frame of depth information, two ranges of depth valuesthat respectively correspond to a foreground region and a backgroundregion of the video data, and responsively define the threshold depth304 to segment the two ranges.

Furthermore, a boundary of the threshold depth 304 may be used to definea head contour. A head contour estimates the outline of a head of auser. A head contour may be generated or obtained via other methods aswell. The head contour may encompass the user's hair or may notencompass the user's hair, depending on the characteristics of theuser's hair as well as the properties of the depth camera used tocapture depth information 302 a. Resultantly, the upper contour 306 mayindicate a hair-background boundary or a forehead-hair boundary. It isassumed that the upper contour 306 takes on one of these two qualitiesand the systems and processes described herein teach a technique foridentifying a hair-region of the user as foreground in view of thisunknown variable.

In some cases, more than one region is identified, wherein the more thanone regions each correspond with depth values that fall within thethreshold depth 304. In order to define the head contour one of the morethan one regions must be selected. In one embodiment, face detection isused to determine the selected region. In another embodiment, a head boxis employed. A head box is a region of pixels that are known to be partof the head. Determining the selected region includes comparing arespective amount of area overlap between each the identified regionsand the head box and determining the selected region to be whichever ofthe identified regions is associated with a greater amount of areaoverlap with the head box.

FIG. 4 depicts a first example generation of an upper contour using ahead contour, in accordance with an embodiment. In particular, FIG. 4depicts a conceptual overview 400 that includes the head of the user,head 402 a and head 402 b. The head 402 a-b may be found in a frame ofvideo data and both the head 402 a and the head 402 b depict the samevideo data. The head 402 a has a head contour 404 shown and the head 402b has an upper contour 406 shown.

In at least one embodiment, obtaining the upper contour 406 of the head402 a (or equivalently head 402 b) of the user includes (i) obtaining ahead contour 404 that estimates an outline of the head 402 a (orequivalently head 402 b) of the user, and (ii) identifying an upperportion of the obtained head contour 404 as being the upper contour 406of the head 402 a (or equivalently head 402 b) of the user. The headcontour 404 may be obtained at least in part using depth data associatedwith the video data, an example of which is described in connection withFIG. 3.

The head contour 404 outlines the user's face but does not include theuser's hair, therefore the identified upper portion of the head contour404 lies between the user's forehead and the user's hair. In turn, theupper contour 406, lies between the user's forehead and the user's hair.

FIG. 5 depicts a second example generation of an upper contour using ahead contour, in accordance with an embodiment. In particular, FIG. 5depicts a conceptual overview 500 that includes the head of the user,head 502 a and head 502 b. The head 502 a-b may be found in a frame ofvideo data and both the head 502 a and the head 502 b depict the samevideo data. The head 502 a has a head contour 504 shown and the head 502b has an upper contour 506 shown.

In at least one embodiment, obtaining the upper contour 506 of the head502 a (or equivalently head 502 b) of the user includes (i) obtaining ahead contour 504 that estimates an outline of the head 502 a (orequivalently head 502 b) of the user, and (ii) identifying an upperportion of the obtained head contour 504 as being the upper contour 506of the head 502 a (or equivalently head 502 b) of the user. The headcontour 504 may be obtained at least in part using depth data associatedwith the video data, an example of which is described in connection withFIG. 3.

The head contour 504 outlines the user's head including the user's hair,therefore the identified upper portion of the head contour 504 liesbetween the user's hair and a background portion of the video data. Inturn, the upper contour 506, lies between the user's hair and abackground portion of the video data. The background portion may bedetermined using the depth data associated with the video data.

The previous portion of this detailed description, with respect to FIGS.4 & 5, highlights two possible upper contour qualities. In FIG. 4 thehead contour 404 does not encompass the user's hair so the upper contour406 is identified as bordering the user's forehead and the user's hair.In FIG. 5 the head contour 504 does encompass the user's hair so theupper contour 506 is identified as bordering the user's hair and abackground in the video data. The background may be determined using thedepth data associated with the video data.

FIG. 6 depicts noisy depth-pixels, in accordance with an embodiment. Inparticular, FIG. 6 depicts a graphical overview 600 that includes depthdata 602 and noisy depth-pixels 604. The depth data 602 may represent afirst frame of depth information. The depth data 602 may be generatedvia a depth camera, 3-D camera, or the like and obtained via acommunication interface. The depth data 602 may alternatively beobtained via a data store.

The noisy depth-pixels 604 are pixels of the depth data 602 that eachhave intermittent depth values. For example, within the depth data 602there are 14 noisy depth-pixels identified as the noisy depth-pixels604. The noisy depth-pixels 604 each had an undetectable depth value ina previous frame or a previous set of frames of depth informationhowever, they each have detectable depth values in the current depthdata 602. A noisy-depth pixel is any pixel that has sporadic depthvalues across a set of frames of depth information. Of course, each ofthe noisy depth pixels corresponds with a pixel of the video data.

FIG. 7 depicts a first set of seed pixels selected from the noisydepth-pixels of FIG. 6, in accordance with an embodiment. In particular,FIG. 7 depicts a graphical overview 700 that includes the depth data 602of FIG. 6 and seed pixels 702. The seed pixels 702 are a set of pixelsselected from the noisy depth-pixels 604 of FIG. 6. The seed pixels 702include every one of the noisy depth-pixels 604. In at least oneembodiment, selecting seed pixels 702 for the flood fill includes (i)identifying noisy depth-pixels 604 of FIG. 6 within the obtained depthdata 602, and (ii) selecting the identified noisy depth-pixels 604 ofFIG. 6 as the seed pixels 702. The seed pixels 702 are to be used forthe flood fill. In some embodiments, color information and depthinformation are stored together in one pixel. In other embodiments,color information and depth information are represented separatelywithin frames of video data and frames of depth data respectively. Theflood fill operates on the video data, therefore using the seed pixels702 is not meant to indicate that the depth-pixels are used as input tothe flood fill, but is meant to indicate that the pixels of the videodata that correspond to the selected depth-pixels are used as input tothe flood fill.

FIG. 8 depicts a second set of seed pixels selected from the noisydepth-pixels of FIG. 6, in accordance with an embodiment. In particular,FIG. 8 depicts a graphical overview 800 that includes the depth data 602of FIG. 6 and seed pixels 802. The seed pixels 802 are a set of pixelsselected from the noisy depth-pixels 604 of FIG. 6. The seed pixels 802do not include every one of the noisy depth-pixels 604 of FIG. 6. In atleast one embodiment, the selected seed pixels 802 have intermittentdepth values similar to a depth value corresponding to the head of theuser. Those noisy depth-pixels 604 that do not have depth values similarto a depth value corresponding to the head of the user are not includedin the set of selected seed pixels 802.

In at least one embodiment, the selected seed pixels 802 are of colorsthat are found in a user-hair-color model. Those noisy depth-pixels 604that are not of colors found in the user-hair-color model are notincluded in the set of selected seed pixels 802. In at least one furtherembodiment, the selected seed pixels 802 are of colors that are found auser-hair-color model at least a threshold number of times. Those noisydepth-pixels 604 that are not of colors found in the user-hair-colormodel at least the threshold number of times are not included in the setof selected seed pixels 802. In at least one embodiment, the selectedseed pixels 802 are of colors that are not found in a background-colormodel. Those noisy depth-pixels 604 that are of colors found in thebackground-color model are not included in the set of selected seedpixels 802. In at least one further embodiment, the selected seed pixels802 are of colors that are found a background-color model no more than athreshold number of times. Those noisy depth-pixels 604 that are ofcolors found in the background-color model more than the thresholdnumber of times are not included in the set of selected seed pixels 802.

FIG. 9 depicts a third set of seed pixels selected from the noisydepth-pixels of FIG. 6, in accordance with an embodiment. In particular,FIG. 9 depicts a graphical overview 900 that includes the depth data 602of FIG. 6, seed pixels 902, and an extended head box 904. The seedpixels 902 are a set of pixels selected from the noisy depth-pixels 604of FIG. 6. The seed pixels 902 do not include every one of the noisydepth-pixels 604 of FIG. 6. In at least one embodiment, the selectedseed pixels 902 are located within an extended head box. Those noisydepth-pixels 604 that are not of colors found in the extended head boxare not included in the set of selected seed pixels 802. Selecting seedpixels for the flood fill may involve any combination of therestrictions discussed with respect to FIGS. 8-9 and the like.

In at least one embodiment, the process further includes initializingthe distance-cost values of the seed pixels to be zero. In at least oneembodiment, the process further includes initializing the distance-costvalues of the seed pixels to be non-zero. In at least one embodiment, afirst set of the selected seed pixels are on an upper contour and asecond set of the selected seed pixels have intermittent depth values.In at least one such embodiment, the process further includesinitializing the distance-cost values of the seed pixels in the firstset to be zero and initializing the distance-cost values of the seedpixels in the second set to be non-zero.

In embodiments wherein the process further includes initializing thedistance-cost values of the seed pixels in the first set to be zero andinitializing the distance-cost values of the seed pixels in the secondset to be non-zero, initializing the distance-cost values of the seedpixels in the second set to be non-zero may be accomplished via avariety of means. In a first example, each seed pixel in the second setis initialized with a common distance-cost value. In a second example,each seed pixel in the second set is initialized with a respectivedistance-cost value.

The non-zero distance-cost value be may be based, at least in part, on adensity of the seed pixels in the second set. The non-zero distance-costvalue be may be based, at least in part, on a distance from a seed pixelin the second set to the upper contour. The non-zero distance-cost valuebe may be based, at least in part, on a distance from a seed pixel inthe second set to the head of the user. The non-zero distance-cost valuebe may be based, at least in part, on a color of the seed pixels in thesecond set. The non-zero distance-cost value be may be based, at leastin part, on a color of the seed pixels in the second set auser-hair-color model.

The following sections of the present disclosure discuss various stepsof the flood fill. FIGS. 10-12 depict example first, second, and thirdsteps of a flood fill respectively. A respective distance-cost value isassigned to each neighbor pixel in a set of neighbor pixels during eachstep of the flood fill. FIG. 10 depicts a current pixel (which in thefollowing example is a seed pixel) and a set of neighbor pixels. FIG. 11also depicts a current pixel and a set of neighbor pixels, however thecurrent pixel in FIG. 11 is one of the neighbor pixels of FIG. 10.Furthermore, FIG. 12 also depicts a current pixel and a set of neighborpixels, however the current pixel in FIG. 12 is one of the neighborpixels of FIG. 11. Performing the flood fill includes performing a floodfill step using a current pixel and a set of neighbor pixels and thenusing at least one, and in some cases each of the neighbor pixels as thecurrent pixel in later flood fill steps. In this manner, the flood fillmay “fill” up an array of pixels with distance-cost values.

In at least one embodiment, performing the flood fill includes (i)identifying a plurality of neighbor pixels of a current pixel, (ii)determining respective step-cost values from the current pixel to eachpixel in the plurality of neighbor pixels, and (iii) assigning eachpixel in the plurality of neighbor pixels a respective distance-costvalue based on a distance-cost value of the current pixel and therespective step-cost values.

In the following example (described with respect to FIGS. 10-12), a stepcost value includes (i) a position-space cost value from a current pixelto a neighbor pixel and (ii) a color-space cost value from the currentpixel to the neighbor pixel. This may be accomplished by employing ageodesic cost that is a combination of the position-space cost valuefrom the current pixel to the neighbor pixel and (ii) the color-spacecost value from the current pixel to the neighbor pixel. It is notedthat each neighbor pixel is an equivalent position-space cost value froma given current pixel.

In other examples, a step cost value includes (i) a position-space costvalue from a current pixel to a neighbor pixel and (ii) a color-spacecost value from a seed pixel to the neighbor pixel. This may beaccomplished by employing a geodesic cost that is a combination of theposition-space cost value from the current pixel to the neighbor pixeland (ii) the color-space cost value from the seed pixel to the neighborpixel. It is noted that each neighbor pixel is an equivalentposition-space distance from a given current pixel.

In other examples, a step cost value includes (i) a position-space costvalue from a current pixel to a neighbor pixel and (ii) a sum ofcolor-space cost values along a flood fill path from a seed pixel to theneighbor pixel. This may be accomplished by employing a geodesic costthat is a combination of the position-space cost value from the currentpixel to the neighbor pixel and (ii) the sum of color-space cost valuesalong the flood fill path from the seed pixel to the neighbor pixel. Itis noted that each neighbor pixel is an equivalent position-spacedistance from a given current pixel.

In other examples, a step cost value includes (i) a position-space costvalue from a current pixel to a neighbor pixel and (ii) a color-spacecost value associated with a user-hair-color model and the neighborpixel. This may be accomplished by employing a geodesic cost that is acombination of the position-space cost value from the current pixel tothe neighbor pixel and (ii) the color-space cost value between theuser-hair-color model and the neighbor pixel. It is noted that eachneighbor pixel is an equivalent position-space distance from a givencurrent pixel.

FIG. 10 depicts an array of pixels including a current pixel and a setof neighbor pixels in a first flood fill step, in accordance with anembodiment. In particular FIG. 10 depicts a graphical overview 1000. Thegraphical overview 1000 depicts an array of pixels 1002 that includes acurrent pixel 1004, and neighbor pixels 1006-1012. The array of pixels1002 represents a set of pixels that is included within the obtainedvideo data. The size of the array of pixels 1002 is five pixels by fivepixels and is not meant to be limiting in any way. Of course, othersized arrays may be used and the choice of a 5×5 array is purely for thesake of visual simplicity.

The current pixel 1004 is a seed pixel. Because the current pixel 1004is a seed pixel, it has an initialized distance-cost value. In at leastone embodiment, performing the flood fill includes identifying aplurality of neighbor pixels (i.e., the neighbor pixels 1006-1012) of acurrent pixel (i.e., the current pixel 1004), (ii) determiningrespective step-cost values from the current pixel 1004 to each pixel inthe plurality of neighbor pixels 1006-1012, and (iii) assigning eachpixel in the plurality of neighbor pixels 1006-1012 a respectivedistance-cost value based on a distance-cost value of the current pixel1004 and the respective step-cost values.

In one example, the current pixel 1004 is black, the neighbor pixel 1006is black, the neighbor pixel 1008 is dark grey, the neighbor pixel 1010is black, and the neighbor pixel 1012 is yellow. The step-cost value ofthe neighbor pixels 1006 and 1010 are small because the neighbor pixels1006 and 1010 are the same color as the current pixel 1004. Thestep-cost value of the neighbor pixel 1008 is also small, but not assmall as the step-cost value of the neighbor pixels 1006 and 1010,because the neighbor pixel 1008 is a similar color to the current pixel1004, but is not the same color as the current pixel 1004. The step-costvalue of the neighbor pixel 1012 is large because the neighbor pixel1012 is a vastly different color than the current pixel 1004.Resultantly, the distance-cost value assigned to the neighbor pixels1006 and 1010 are the same and the smallest of this example. Thedistance-cost value assigned to the neighbor pixel 1008 is larger thanthe distance-cost value assigned to the neighbor pixels 1006 and 1010.The distance-cost value assigned to the neighbor pixel 1012 is thelargest of this example.

FIG. 11 depicts an array of pixels including a current pixel and a setof neighbor pixels in a second flood fill step, in accordance with anembodiment. In particular FIG. 11 depicts a graphical overview 1100. Thegraphical overview 1100 depicts the array of pixels 1002 of FIG. 10. Inthis second step of the flood fill, the neighbor pixel 1008 of FIG. 10is now described as a current pixel 1008 and the current pixel 1004 ofFIG. 10 is now described as a neighbor pixel 1004. As depicted in thegraphical overview 1100, the current pixel 1008 has neighbor pixels1102-1106 and 1004. The current pixel 1008 is not a seed pixel, and ithas a non-zero distance-cost value (which was assigned in the first stepof the flood fill as depicted in FIG. 10).

Furthering the example discussed with respect to FIG. 10, the currentpixel 1008 is dark grey, the neighbor pixel 1102 is black, the neighborpixel 1104 is dark grey, the neighbor pixel 1106 is black, and theneighbor pixel 1004 is black. The step-cost value of the neighbor pixel1104 is small because the neighbor pixel 1104 is the same color as thecurrent pixel 1008. The step-cost values of the neighbor pixels 1102,1106, and 1004 are also small, but not as small as the step-cost valueof the neighbor pixel 1104, because the neighbor pixels 1102, 1106, and1004 are a similar color to the current pixel 1008, but the neighborpixels 1102, 1106, and 1004 are not the same color as the current pixel1008. Resultantly, the distance-cost value assigned to the neighborpixel 1104 is small in this example, however it must be greater than thedistance-cost value assigned to the current pixel 1008 because thedistance-cost value is a running sum. The distance-cost value assignedto the neighbor pixels 1102 and 1106 is larger than the distance-costvalue assigned to the neighbor pixels 1104. The distance-cost valueassigned to the neighbor pixel 1004 is unchanged from the last step ofthe flood fill because it is larger than the previously assigneddistance-cost value (i.e., the flood fill process does not re-assigndistance cost values if the new value is larger than the old value).This is consistent with embodiments wherein, performing the flood fillincludes (i) determining a minimum distance-cost value from at least oneof the selected seed pixels to a candidate pixel (e.g., the neighborpixel 1110), and (ii) assigning the candidate pixel (e.g., the neighborpixel 1110) a distance-cost value that is the determined minimumdistance-cost value.

FIG. 12 depicts an array of pixels including a current pixel and a setof neighbor pixels in a third flood fill step, in accordance with anembodiment. In particular FIG. 12 depicts a graphical overview 1200. Thegraphical overview 1200 depicts the array of pixels 1002 of FIG. 10. Inthis third step of the flood fill, the neighbor pixel 1106 of FIG. 11 isnow described as a current pixel 1106 and the current pixel 1008 of FIG.11 is now described as a neighbor pixel 1008. As depicted in thegraphical overview 1200, the current pixel 1106 has neighbor pixels1008, 1202, 1204, and 1010. The current pixel 1106 is not a seed pixel,and it has a non-zero distance-cost value (which was assigned in thesecond step of the flood fill as depicted in FIG. 11).

Furthering the example discussed with respect to FIGS. 10 and 11, thecurrent pixel 1106 is black, the neighbor pixel 1008 is dark grey, theneighbor pixel 1202 is dark grey, the neighbor pixel 1204 is dark grey,and the neighbor pixel 1010 is black. The step-cost value of theneighbor pixel 1010 is small because the neighbor pixel 1010 is the samecolor as the current pixel 1106. The step-cost values of the neighborpixels 1008, 1202, and 1204 are also small, but not as small as thestep-cost value of the neighbor pixel 1010, because the neighbor pixels1008, 1202, and 1204 are a similar color to the current pixel 1106, butthe neighbor pixels 1008, 1202, and 1204 are not the same color as thecurrent pixel 1106. Resultantly, the distance-cost values assigned tothe neighbor pixels 1008 and 1010 are unchanged. The distance-costvalues assigned to the neighbor pixels 1202 and 1204 are larger than thedistance-cost values previously assigned to the neighbor pixels 1008 and1010. The distance-cost values assigned to the neighbor pixel 1008 and1010 are unchanged from the last step of the flood fill because they arelarger than the previously assigned distance-cost values. This isconsistent with embodiments wherein, performing the flood fill includes(i) determining a minimum distance-cost value from at least one of theselected seed pixels to a candidate pixel, and (ii) assigning thecandidate pixel a distance-cost value that is the determined minimumdistance-cost value.

FIG. 13 depicts an array of pixels including two seed pixels and a givenpixel, in accordance with an embodiment. In particular, FIG. 13 depictsa graphical overview 1300. The graphical overview 1300 depicts an arrayof pixels 1302 that includes pixels 1304-1346. The array of pixels 1302represents a set of pixels that is included within the obtained videodata. The size of the array of pixels 1302 is ten pixels by six pixelsand is not meant to be limiting in any way. Of course, other sizedarrays may be used and the choice of a 10×6 array is purely for the sakeof visual simplicity.

The pixels 1304 and 1330 are seed pixels. The pixel 1314 is apixel-of-interest in the given example. It may be referred to as a givenpixel or a current pixel.

Depicted in the graphical overview 1300, there are three unique floodfill paths that originate from a seed pixel and terminate at the givenpixel (i.e., the pixel 1314). A first path is defined by the pixels1304-1313. A second path is defined by the pixels 1304-1306, 1316-1328,and 1316. A third path is defined by the pixels 1330-1346 and 1314.

Each of the three flood fill paths is an example of a set of flood fillsteps. The first path has a position-space cost value of 5 pixels fromthe seed pixel 1304 to the given pixel 1314 (i.e., it costs 5 flood fillsteps to get from the seed pixel 1304 to the given pixel 1314 along thefirst flood fill path or the flood fill must traverse 5 pixels to getfrom the seed pixel 1304 to the given pixel 1314 along the first floodfill path). The second path has a position-space cost value of 9 pixelsfrom the seed pixel 1304 to the given pixel 1314. The third path has aposition-space cost value of 9 from the seed pixel 1330 to the givenpixel 1314.

The color-space cost value associated with the given pixel 1314 may becalculated in a plurality of ways. In at least one embodiment, thedistance-cost value of the given pixel (i.e., the pixel 1314) includes(i) a position-space cost value from a seed pixel (either the seed pixel1304 or the seed pixel 1330) to the given pixel 1314 and (ii) acolor-space cost value from the seed pixel (either the seed pixel 1304or the seed pixel 1330) to the given pixel 1314.

In at least one embodiment, the distance-cost value of a given pixel isa geodesic cost from a seed pixel (either the seed pixel 1304 or theseed pixel 1330) to the given pixel 1314. The geodesic cost is acombination of a position-space cost value from the seed pixel (eitherthe seed pixel 1304 or the seed pixel 1330) to the given pixel 1314 and(ii) a color-space cost value from the seed pixel (either the seed pixel1404 or the seed pixel 1330) to the given pixel 1314.

In at least one embodiment, performing the flood fill includes (i)determining a minimum distance-cost value from at least one of theselected seed pixels (either the seed pixel 1304 or the seed pixel 1330)to a candidate pixel (i.e., the given pixel 1314), and (ii) assigningthe candidate pixel a distance-cost value that is the determined minimumdistance-cost value.

In a first possibility, the color-space cost value associated with thegiven pixel 1314 is seed pixel and path independent. In the firstpossibility, the color-space cost value is based on a user-hair-colormodel and the given pixel 1314. The color-space cost value may becalculated by using the value of the user-hair-color model at the colorof the given pixel 1314. The color-space cost value may be calculatedusing a difference between a mean color of the user-hair-color model andthe given pixel 1314. The color-space cost value may be calculated usinga difference between a mode color of the user-hair-color model and thegiven pixel 1314. The color-space cost value may be calculated using adifference between a median color of the user-hair-color model and thegiven pixel 1314. Of course, other techniques (which may or may notemploy statistical parameters associated with the user-hair-color model)may be used to calculate the color-space cost value of the given pixel1314.

In a second possibility, the color-space cost value associated with thegiven pixel 1314 is path independent and seed pixel dependent. In thesecond possibility, the color-space cost value is based on a colordifference between a seed pixel and the given pixel 1314. In FIG. 13there are two possible color-space cost values in the context of thesecond possibility (one associated with the seed pixel 1304 and anotherassociated with the seed pixel 1330).

In a third possibility, the color-space cost value associated with thegiven pixel 1314 is path dependent (and as a result of logicaldependency, seed pixel dependent). In the third possibility, thecolor-space cost value is a summation of color differences accumulatedalong a given path. In FIG. 13 there are three possible color-space costvalues in the context of the third possibility (corresponding to thethree paths—two associated with the seed pixel 1304 and one associatedwith the seed pixel 1330). In each step of the flood fill, a step-costvalue associated with a neighbor pixel is calculated as described withrespect to FIG. 10-12.

In the second and third possibilities the assigned distance-cost valueis the smallest distance-cost value of the available options. In atleast one embodiment, determining the minimum distance-cost valueincludes comparing a current distance-cost value corresponding with acurrent path (e.g., the second path) to a prior distance-cost valuecorresponding with a prior path (e.g., the first path). In at least onesuch embodiment, the current path and the prior path originate from acommon seed pixel (e.g., the first and second paths and the seed pixel1304). In at least one other such embodiment, the current path and theprior path originate from different seed pixels (e.g., the first andthird paths and the seed pixels 1304 and 1330).

In at least one embodiment, performing the flood fill comprisesperforming the flood fill along a plurality of flood-fill paths. In atleast one such embodiment the process further includes terminating theflood fill along a current flood-fill path in response to at least onetermination criteria. The termination criteria includes a current pixelnot being a user-hair color according to a user-hair-color model, thecurrent pixel being a background color according to a background-colormodel, a distance-cost value of the current pixel being greater than adistance-cost threshold, and a step-cost value of the current pixelbeing greater than a step-cost threshold. Of course many othertermination criteria could be employed as well such as aposition-space-cost value of the current pixel being greater than aposition-space-cost threshold and a color-space-cost value of thecurrent pixel being greater than a color-space-cost threshold.

FIG. 14 depicts an example computing and communication device (CCD), inaccordance with an embodiment. In the embodiment that is depicted inFIG. 14, an example CCD 1400 includes a communication interface 1402, aprocessor 1404, and data storage 1406 containing instructions 1408executable by the processor 1404 for causing the CCD 1400 to carry out aset of functions, which may include those functions described above inconnection with FIG. 1. As a general matter, the example CCD 1400 ispresented as an example system that could be programmed and configuredto carry out the functions described herein.

The communication interface 1402 may include one or morewireless-communication interfaces (for communicating according to, e.g.,APCO P25, TETRA, DMR, LTE, Wi-Fi, NFC, Bluetooth, and/or one or moreother wireless-communication protocols) and/or one or morewired-communication interfaces (for communicating according to, e.g.,Ethernet, USB, eSATA, IEEE 1394, and/or one or more otherwired-communication protocols). As such, the communication interface1402 may include any necessary hardware (e.g., chipsets, antennas,Ethernet cards, etc.), any necessary firmware, and any necessarysoftware for conducting one or more forms of communication with one ormore other entities as described herein. The processor 1404 may includeone or more processors of any type deemed suitable by those of skill inthe relevant art, some examples including a general-purposemicroprocessor and a dedicated digital signal processor (DSP).

The data storage 1406 may take the form of any non-transitorycomputer-readable medium or combination of such media, some examplesincluding flash memory, read-only memory (ROM), and random-access memory(RAM) to name but a few, as any one or more types of non-transitorydata-storage technology deemed suitable by those of skill in therelevant art could be used. As depicted in FIG. 14, the data storage1406 contains program instructions 1408 executable by the processor 1404for carrying out various functions, and also contains operational data1410, which could include any one or more types of data stored and/oraccessed by the example CCD 1400 during operation. In embodiments inwhich a computing system such as the example CCD 1400 is arranged,programmed, and configured to carry out processes such as the exampleprocess that is described above in connection with FIG. 1, the programinstructions 1408 are executable by the processor 1404 for carrying outthose functions; in instances where other entities described herein havea structure similar to that of the example CCD 1400, the respectiveprogram instructions 1408 for those respective devices are executable bytheir respective processors 1404 to carry out functions respectivelyperformed by those devices.

If present, the user interface 1412 may include one or more inputdevices (a.k.a. components and the like) and/or one or more outputdevices (a.k.a. components and the like). With respect to input devices,the user interface 1412 may include one or more touchscreens, buttons,switches, microphones, and the like. With respect to output devices, theuser interface 1412 may include one or more displays, speakers, lightemitting diodes (LEDs), and the like. Moreover, one or more components(e.g., an interactive touchscreen-and-display component) of the userinterface 1412 could provide both user-input and user-outputfunctionality. And certainly other user-interface components could beused in a given context, as known to those of skill in the art.Furthermore, the CCD 1400 may include one or more video cameras, depthcameras, 3-D cameras, infrared-visible cameras, light-field cameras or acombination thereof.

In the foregoing specification, specific embodiments have beendescribed. However, one of ordinary skill in the art appreciates thatvarious modifications and changes can be made without departing from thescope of the invention as set forth in the claims below. Accordingly,the specification and figures are to be regarded in an illustrativerather than a restrictive sense, and all such modifications are intendedto be included within the scope of present teachings.

The benefits, advantages, solutions to problems, and any element(s) thatmay cause any benefit, advantage, or solution to occur or become morepronounced are not to be construed as a critical, required, or essentialfeatures or elements of any or all the claims. The invention is definedsolely by the appended claims including any amendments made during thependency of this application and all equivalents of those claims asissued.

Moreover in this document, relational terms such as first and second,top and bottom, and the like may be used solely to distinguish oneentity or action from another entity or action without necessarilyrequiring or implying any actual such relationship or order between suchentities or actions. The terms “comprises,” “comprising,” “has,”“having,” “includes,” “including,” “contains,” “containing,” or anyother variation thereof, are intended to cover a non-exclusiveinclusion, such that a process, method, article, or apparatus thatcomprises, has, includes, contains a list of elements does not includeonly those elements but may include other elements not expressly listedor inherent to such process, method, article, or apparatus. An elementpreceded by “comprises . . . a,” “has . . . a,” “includes . . . a,”“contains . . . a” does not, without more constraints, preclude theexistence of additional identical elements in the process, method,article, or apparatus that comprises, has, includes, contains theelement. The terms “a” and “an” are defined as one or more unlessexplicitly stated otherwise herein. The terms “substantially,”“essentially,” “approximately,” “about,” or any other version thereof,are defined as being close to as understood by one of ordinary skill inthe art, and in one non-limiting embodiment the term is defined to bewithin 1%, in another embodiment within 5%, in another embodiment within1% and in another embodiment within 0.5%. The term “coupled” as usedherein is defined as connected, although not necessarily directly andnot necessarily mechanically. A device or structure that is “configured”in a certain way is configured in at least that way, but may also beconfigured in ways that are not listed.

It will be appreciated that some embodiments may be comprised of one ormore generic or specialized processors (or “processing devices”) such asmicroprocessors, digital signal processors, customized processors andfield programmable gate arrays (FPGAs) and unique stored programinstructions (including both software and firmware) that control the oneor more processors to implement, in conjunction with certainnon-processor circuits, some, most, or all of the functions of themethod and/or apparatus described herein. Alternatively, some or allfunctions could be implemented by a state machine that has no storedprogram instructions, or in one or more application specific integratedcircuits (ASICs), in which each function or some combinations of certainof the functions are implemented as custom logic. Of course, acombination of the two approaches could be used.

Moreover, an embodiment can be implemented as a computer-readablestorage medium having computer readable code stored thereon forprogramming a computer (e.g., comprising a processor) to perform amethod as described and claimed herein. Examples of suchcomputer-readable storage mediums include, but are not limited to, ahard disk, a CD-ROM, an optical storage device, a magnetic storagedevice, a ROM (Read Only Memory), a PROM (Programmable Read OnlyMemory), an EPROM (Erasable Programmable Read Only Memory), an EEPROM(Electrically Erasable Programmable Read Only Memory) and a Flashmemory. Further, it is expected that one of ordinary skill,notwithstanding possibly significant effort and many design choicesmotivated by, for example, available time, current technology, andeconomic considerations, when guided by the concepts and principlesdisclosed herein will be readily capable of generating such softwareinstructions and programs and ICs with minimal experimentation.

The Abstract of the Disclosure is provided to allow the reader toquickly ascertain the nature of the technical disclosure. It issubmitted with the understanding that it will not be used to interpretor limit the scope or meaning of the claims. In addition, in theforegoing Detailed Description, it can be seen that various features aregrouped together in various embodiments for the purpose of streamliningthe disclosure. This method of disclosure is not to be interpreted asreflecting an intention that the claimed embodiments require morefeatures than are expressly recited in each claim. Rather, as thefollowing claims reflect, inventive subject matter lies in less than allfeatures of a single disclosed embodiment. Thus the following claims arehereby incorporated into the Detailed Description, with each claimstanding on its own as a separately claimed subject matter.

1. A method comprising: obtaining video data depicting a head of a user;obtaining depth data associated with the video data; selecting seedpixels for a hair-identification flood fill for identifying pixelsdepicting hair of the head of the user, the seed pixels selected atleast in part by using the obtained depth data; performing thehair-identification flood fill from the selected seed pixels, thehair-identification flood fill assigning respective distance-cost valuesto pixels of the video data based on respective position-space-costvalues and respective color-space-cost values; and identifying a personaof the user from the video data based at least in part on the respectivedistance-cost values assigned by the hair-identification flood fill. 2.The method of claim 1, wherein selecting seed pixels for thehair-identification flood fill further comprises selecting seed pixelsfor the hair-identification flood fill at least in part by using thevideo data.
 3. The method of claim 1, wherein selecting seed pixels forthe hair-identification flood fill comprises: obtaining a head contourthat estimates an outline of the depicted head of the user, the headcontour being based at least in part on the depth data associated withthe video data; and selecting pixels that are on an upper contour asseed pixels for the hair-identification flood fill, wherein the uppercontour is an upper portion of the head contour.
 4. The method of claim3, wherein the selected seed pixels are equally distributed along theupper contour.
 5. The method of claim 3, wherein the selected seedpixels are of colors that are found in a user-hair-color model.
 6. Themethod of claim 1, wherein selecting seed pixels for thehair-identification flood fill comprises: identifying pixels havingnoisy depth values over a series of frames; and selecting the identifiednoisy depth-pixels as seed pixels for the hair-identification floodfill.
 7. The method of claim 6, wherein the selected seed pixels arelocated within an extended head box.
 8. The method of claim 6, whereinthe selected seed pixels have intermittent depth values that are withina threshold tolerance of a depth value corresponding to the head of theuser.
 9. The method of claim 6, wherein the selected seed pixels are ofcolors that are found in a user-hair-color model.
 10. The method ofclaim 1, wherein a first set of the selected seed pixels are on an uppercontour and a second set of the selected seed pixels have noisy depthvalues over a series of frames, the method further comprising:initializing the distance-cost values of the seed pixels in the firstset to be zero; and initializing the distance-cost values of the seedpixels in the second set to be non-zero.
 11. The method of claim 1,wherein a distance-cost value of a given pixel comprises (i) aposition-space cost value from a seed pixel to the given pixel and (ii)a color-space cost value from the seed pixel to the given pixel.
 12. Themethod of claim 1, wherein performing the hair-identification flood fillcomprises: identifying a plurality of neighbor pixels of a currentpixel; determining respective step-cost values from the current pixel toeach pixel in the plurality of neighbor pixels; and assigning each pixelin the plurality of neighbor pixels a respective distance-cost valuebased on a distance-cost value of the current pixel and the respectivestep-cost values.
 13. The method of claim 1, wherein performing thehair-identification flood fill comprises: determining a minimumdistance-cost value from at least one of the selected seed pixels to acurrent pixel; and assigning the current pixel a distance-cost valuethat is the determined minimum distance-cost value.
 14. The method ofclaim 13, wherein determining a minimum distance-cost value comprises:comparing a current distance-cost value corresponding with a currentflood-fill path to a prior distance-cost value corresponding with aprior flood-fill path.
 15. The method of claim 14, wherein the currentflood-fill path and the prior flood-fill path originate from a commonseed pixel.
 16. The method of claim 14, wherein the current flood-fillpath and the prior flood-fill path originate from different seed pixels.17. The method of claim 1, wherein performing the hair-identificationflood fill comprises performing the hair-identification flood fill alonga plurality of flood-fill paths, the method further comprising:terminating the hair-identification flood fill along a currentflood-fill path in response to at least one termination criteria, thetermination criteria comprising: a current pixel not being a user-haircolor according to a user-hair-color model; the current pixel being abackground color according to a background-color model; a distance-costvalue to the current pixel being greater than a distance-cost threshold;and a step-cost value to the current pixel being greater than astep-cost threshold.
 18. The method of claim 1, wherein identifying thepersona of the user from the video data based at least in part on therespective distance-cost values assigned by the hair-identificationflood fill comprises classifying pixels of the video data as foregroundbased at least in part on the assigned distance-cost values.
 19. Themethod of claim 1, wherein identifying the persona of the user from thevideo data based at least in part on the respective distance-cost valuesassigned by the hair-identification flood fill comprises assigningpixels of the video data foreground-likelihood values based at least inpart on the assigned distance-cost values.
 20. A system comprising: acommunication interface; a processor; and data storage containinginstructions executable by the processor for causing the system to carryout a set of functions, the set of functions including: obtaining videodata depicting a head of a user; obtaining depth data associated withthe video data; selecting seed pixels for a hair-identification floodfill for identifying pixels depicting hair of the head of the user, theseed pixels selected at least in part by using the depth data;performing the hair-identification flood fill from the selected seedpixels, the hair-identification flood fill assigning respectivedistance-cost values to pixels of the video data based on respectiveposition-space cost values and respective color-space cost values; andidentifying a persona of the user from the video based at least in parton the respective distance-cost values assigned by thehair-identification flood fill.